[{"slug":"always-specify-user-agent-in-fetch","frontmatter":{"content":"\nWhile performing audits it is a superpower to be able to fetch the website at Edge(e.g. Cloudflare Workers) and modify the HTML on the fly to check if a performance idea would work or not. But sometimes, a website can give 403. Usually, I think the website has some IP restrictions of a certain country, but that shouldn't be the case here as the Edge node would be very close to where the actual visitor is accessing the website. \n\nThe reason can sometimes very simply be not providing a proper User-Agent, which happened today with me while I scratched my head for a long time. \n\nAlways, specify User-Agent while doing a fetch\n\n```jsx\n// A website that gives 404 if UA is not provided or a common browser UA isn't used.\nfetch('https://example.com', {\n headers: {\n 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Safari/537.36'\n }\n }\n );\n```","data":{"category":"Cloudflare Workers, TIL","creationTs":1624618980000,"date":"June 25, 2021 1:03 PM","lastModified":"September 9, 2021 3:10 PM","lastModifiedTs":1631193000000,"published":"Yes","title":"Always specify User-Agent in fetch"},"isEmpty":false,"excerpt":"","category":["Cloudflare Workers","TIL"],"creationTs":1624618980000,"date":"June 25, 2021","lastModified":"September 9, 2021 3:10 PM","lastModifiedTs":1631193000000,"published":"Yes","title":"Always specify User-Agent in fetch","updated":"September 9, 2021"},"content":"

While performing audits it is a superpower to be able to fetch the website at Edge(e.g. Cloudflare Workers) and modify the HTML on the fly to check if a performance idea would work or not. But sometimes, a website can give 403. Usually, I think the website has some IP restrictions of a certain country, but that shouldn't be the case here as the Edge node would be very close to where the actual visitor is accessing the website.

The reason can sometimes very simply be not providing a proper User-Agent, which happened today with me while I scratched my head for a long time.

Always, specify User-Agent while doing a fetch

// A website that gives 404 if UA is not provided or a common browser UA isn't used.\nfetch('https://example.com', {\n    headers: {\n      'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Safari/537.36'\n      }\n    }\n  );\n

\n"}]

Category: Cloudflare Workers